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SOFT PICTURE/GRAPHICS CLASSIFICATION 
SYSTEM AND METHOD 

Background of the Invention 

[0001] The present invention relates to image processing. 

It finds particular application in conjunction with classification of images between 
natural pictures and synthetic graphics, and will be described with particular 
reference thereto. However, it is to be appreciated that the present invention is also 
amenable to other like applications. 

[0002] During the past several decades, products and 

services such as TVs, video monitors, photography, motion pictures, copying 
devices, magazines, brochures, newspapers, etc. have steadily evolved from 
monochrome to color. With the increasing use of color products and services, there 
is a growing demand for "brighter" and more "colorful" colors in several 
applications. Due to this growing demand, display and printing of color imagery 
that is visually pleasing has become a very important topic. In a typical color copier 
application, the goal is to render the scanned document in such a way that it is most 
pleasing to the user. 

[0003] Natural pictures differ from synthetic graphics in 

many aspects, both in terms of visual perception and image statistics. Synthetic 
graphics are featured with smooth regions separated by sharp edges. On the 
contrary, natural pictures are often noisier and the region boundaries are less 
prominent. In processing scanned images, it is sometime beneficial to distinguish 
images from different origins (e.g., synthetic graphics or natural pictures), however, 
the origin or "type" information about a scanned image is usually unavailable. The 
"type" information should be automatically extracted from the scanned image. This 
"type" information is then used in further processing of the images. High-level 



image classification can be achieved by analysis of low-level image attributes geared 
for the particular classes. Coloring schemes (e.g., gamut-mapping, or filtering 
algorithms) are tailored for specific types of images to obtain quality reproduction. 

Once an image has been identified as a graphics image, further identification of 
image characteristics can be used to fine-tune the coloring schemes for more 
appealing reproductions. The most prominent characteristics of a graphics image 
include patches or areas of the image with uniform color and areas with uniformly 
changing colors. These areas of uniformly changing color are called sweeps. 

[0004] Picture/graphics classifiers have been developed to 

differentiate between a picture image and a graphics image by analyzing low-level 
image statistics. For example, U.S. Patent No. 5,767,978 to Revankar et al. 
discloses an adaptable image segmentation system for differentially rendering black 
and white and/or color images using a plurality of imaging techniques. An image 
is segmented according to classes of regions that may be rendered according to the 
same imaging techniques. Image regions may be rendered according to a three- 
class system (such as traditional text, graphic, and picture systems), or according 
to more than three (3) image classes. In addition, only two (2) image classes may 
be required to render high quality draft or final output images. The image 
characteristics that may be rendered differently from class to class may include half 
toning, colorization and other image attributes. 

[0005] Graphics are typically generated using a limited 

number of colors, usually containing only a few areas of uniform colors. On the 
other hand, natural pictures are more noisy, containing smoothly varying colors. 

A picture/graphics classifier can analyze the colors to distinguish between picture 
and graphics images. 

[0006] Graphics images contain several areas of uniform 

color, lines drawings, text, and have very sharp, prominent, long edges. On the 
other hand, natural pictures are very noisy and contain short broken edges. A 
picture/graphics classifier can analyze statistics based on edges to distinguish 
between picture and graphics images. 



[0007] Classifiers that can be used to solve a certain 

classification problem include statistical, structural, neural networks, fuzzy logic, 
and machine learning classifiers. Several of these classifiers are available in public 
domain and commercial packages. However, no single classifier seems to be highly 
successful in dealing with complex real world problems. Each classifier has its own 
weaknesses and strengths. 

[0008] The picture/graphics classification methods described 

above each use features of the image to make a "binary" classification decision (i.e., 
picture or graphics). The binary classification result is then used to "switch" 
between image processing functions. However, using the current set of features 
and the binary classification scheme, the classification accuracy, as tested on large 
image sets, is not perfect. Even with improved features and the binary classification 
scheme, it may not be possible to achieve perfect classification. In fact, there are 
images for which a clear classification cannot even be made by a human observer. 
Under such circumstances, the binary decision is often wrong, and could lead to 
objectionable image artifacts. 

[0009] U.S. Patent No. 5,778,156 to Schweid et al 

discloses an improved method of image processing utilizing a fuzzy logic 
classification process. The disclosure includes a system and method to 
electronically image process a pixel belonging to a set of digital image data with 
respect to a membership of the pixel in a plurality of image classes. This process 
uses classification to determine a membership value for the pixel for each image 
class and generates an effect tag for the pixel based on the fuzzy classification 
determination. The pixel is image processed based on the membership vector of the 
pixel. The image processing may include screening and filtering. The screening 
process screens the pixel by generating a screen value according to a position of the 
pixel in the set of digital image data; generating a screen amplitude weighting value 
based on the values in the membership vector for the pixel; multiplying the screen 
value and the screen amplitude weighting value to produce a modified screen value; 
and adding the modified screen value to the pixel of image data. The filtering 
process filters the pixel by low-pass filtering the pixel; high-pass filtering the pixel; 
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non-filtering the pixel; multiplying each filtered pixel by a gain factor based on the 
values in the membership vector associated with the pixel; and adding the products 
to produce a filtered pixel of image data. 

[0010] The present invention contemplates new and 

improved methods for classifying images that overcome the above-referenced 
problems and others. 

Summary of the Invention 

[0011] In accordance with one aspect of the present 

invention, a method for classification of an image is provided. The method is 
comprised of: a) extracting a plurality of features from an input image; and b) 
classifying the input image in picture or graphics classes using a combination of two 
or more of the extracted features. 

[0012] In accordance with another aspect of the present 

invention, a method for evaluating the confidence level of the classification of an 
image is provided. The method is comprised of: a) extracting a plurality of features 
from an input image; b) classifying the input image in picture or graphics classes 
using at least one of the extracted features to; and c) determining the confidence 
level of the classification using a combination of two or more of the extracted 
features. 

[0013] In accordance with another aspect of the present 

invention, a method for classification of an input image in natural picture or 
synthetic graphics classes is provided. The method is comprised of: a) extracting 
one or more spatial gray-level dependence texture features from the input image; 

b) processing each extracted feature using an algorithm associated with the feature; 

c) comparing the result of each feature algorithm to one or more previously selected 
thresholds; and d) if, according to previously determined rules, any comparison is 
determinative of the class of the input image, classifying the input image in either 
the natural picture or synthetic graphics class according to the previously 
determined rules, otherwise indicating the result is indeterminate. 
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[0014] In accordance with another aspect of the present 

invention, another method for classification of an input image in natural picture or 
synthetic graphics classes is provided. The method is comprised of: a) extracting 
one or more color discreteness features from the input image; b) processing each 
extracted feature using an algorithm associated with the feature; c) comparing the 
result of each feature algorithm to one or more previously selected thresholds; and 
d) if, according to previously determined rules, any comparison is determinative of 
the class of the input image, classifying the input image in either the natural picture 
or synthetic graphics classes according to the previously determined rules, 
otherwise indicating the result is indeterminate. 

[0015] In accordance with another aspect of the present 

invention, another method for classification of an input image in a synthetic graphics 
class is provided. The method is comprised of: a) extracting one or more edge 
features from the input image; b) processing each extracted feature using an 
algorithm associated with the feature; c) comparing the result of each feature 
algorithm to one or more previously selected thresholds; and d) if, according to 
previously determined rules, any comparison is determinative of the class of the 
input image, classifying the input image in either the natural picture or synthetic 
graphics classes according to the previously determined rules, otherwise indicating 
the result is indeterminate. 

[0016] In accordance with another aspect of the present 

invention, another method for classification of an input image in natural picture or 
synthetic graphics classes is provided. The method is comprised of: a) extracting 
a plurality of features from an input image; and b) processing two or more extracted 
features using a neural network to classify the input image in either natural picture 
or synthetic graphics classes. 

[0017] In accordance with another aspect of the present 

invention, an image processing system for producing an output image associated 
with an input image based on classification of the input image is provided. The 
system is comprised of: a feature extractor for extracting a plurality of features from 
the input image; a binary classifier for classifying the input image in natural picture 
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or synthetics graphics classes using a combination of any two or more of the 
extracted features; a picture processing module for processing the input image using 
picture image processing functions; a graphics processing module for processing the 
input image using graphics image processing functions; and a switch for routing the 
input image for image processing by the picture processing module or the graphics 
processing module based on the classification of the input image by the binary 
classifier in either natural picture and synthetic graphics classes. 

[0018] In accordance with another aspect of the present 

invention, a method for classification of areas of an input image in picture, graphics, 
or fuzzy classes is provided. The method is comprised of: a) extracting a plurality 
of features from an input image; and b) processing two or more extracted features 
using a soft classifier to classify areas of the input image in either picture, graphics, 
or fuzzy classes. 

[0019] In accordance with another aspect of the present 

invention, an image processing system for producing an output image associated 
with an input image based on classification of areas of the input image is provided. 
The system is comprised of: a feature extractor for extracting a plurality of features 
from the input image; a soft classifier for classifying areas of the input image in 
picture, graphics, or fuzzy classes using a combination of any two or more of the 
extracted features; a plurality of image processing modules for providing a plurality 
of image processing functions; and a blender for blending the image processing 
functions from the image processing modules, said blending based on the 
classification of areas of the input image by the soft classifier. 

[0020] One advantage of the present invention is that an 

input image is classified as either a natural picture or synthetic graphics with less 
error than prior classifiers by using new features for classification. 

[002 1 ] Another advantage of the present invention is that an 

input image is classified as either a natural picture or synthetic graphics with less 
error than prior classifiers by using combinations of features for classification. 



[0022] Another advantage of the present invention is that an 

input image is classified by a "soft" classifier using new features and combinations 
of features to classify areas of the image as either picture, graphics, or fuzzy classes. 

[0023] Another advantage of the present invention is that 

the "soft" classifier is able to predict a confidence level for picture and graphics 
image classification. 

[0024] Another advantage of the present invention is that 

image processing functions are blended in conjunction with picture, graphics, and 
fuzzy classifications of image areas by the "soft" classifier to produce a more 
desirable output image than prior image processing systems. 

[0025] Still further advantages and benefits of the present 

invention will become apparent to those of ordinary skill in the art upon reading and 
understanding the following detailed description of the preferred embodiments. 

Brief Description of the Drawings 

[0026] The invention may take form in various components 

and arrangements of components, and in various steps and arrangements of steps. 
The drawings are only for purposes of illustrating preferred embodiments and are 
not to be construed as limiting the invention. 

[0027] FIG. 1 is a flowchart of an image classification 

process using SGLD texture features in accordance with an embodiment of the 
present invention; 

[0028] FIG. 2 is a flowchart of the SGLD matrix 

initialization and construction process in accordance with an embodiment of the 
present invention; 

[0029] FIG. 3 is a flowchart of an image classification 

process using color discreteness features in accordance with an embodiment of the 
present invention; 

[0030] FIG. 4 is a flowchart of an image classification 

process using edge features in accordance with an embodiment of the present 
invention; 
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[0031] FIG. 5 is a flowchart of an image classification 

process using a combination of SOLD texture features, color discreteness features, 
and edge features in accordance with an embodiment of the present invention; 

[0032] FIG. 6 is a block diagram of an image processing 

system using a "binary" image classification process (i.e., classification of images 
between picture or graphics classes); and 

[0033] FIG. 7 is a block diagram of an image processing 

system using a "soft" image classification process (i.e., classification of image areas 
between picture, graphics, or fuzzy classes) and an associated process for blending 
image processing functions based on the classification. 



Detailed Description of the Preferred Embodiments 

[0034] Spatial gray-level dependence (SGLD) techniques 

for image analysis are well known. SGLD feature extraction creates a two- 
dimensional histogram that measures first and second-order statistics of an image. 
These features are captured in SGLD matrices. This was originally proposed for 
texture analysis of multi-level images. Additionally, since texture features 
distinguish natural pictures from synthetic graphics, SGLD techniques can be 
applied to picture/graphics classification of images. A picture/graphics classifier can 
be created with algorithms that analyze the texture features captured in SGLD 
matrices. Using the SGLD texture features, the classifier works to determine 
whether a scanned image is a natural picture or synthetic graphics. Furthermore, 
in color images, the luminance component typically contains enough information to 
determine the origin of the image. Therefore, an SGLD matrix that captures the 
luminance component of an image and a picture/graphics classifier using the 
luminance component from the matrix in a classification algorithm can determine 
whether the image is a natural picture or synthetic graphics. 

[0035] With reference to FIG. 1, a flowchart of an image 

classification process using SGLD texture features 100 in accordance with an 
embodiment of the present invention is shown. Generally, the classification process 
filters an input image to smooth out halftones, builds an SGLD matrix from the 
smoothed image, extracts texture features from the matrix, and performs an 
algorithm to determine whether the image is a natural picture or synthetic graphics 
based on one (1) or more of the texture features. 

[0036] More specifically, the process 100 begins with an 

input image 102. The image is processed using a low-pass filter 104 (e.g., a W x 
W averaging filter) to smooth the luminance component and reduce any halftone 
noise. The SGLD matrix is basically a GL x GL two-dimensional histogram, where 
GL is the number of gray levels (e.g., 256). The SGLD matrix is generated by first 
performing an initialization (e.g., set to zero) 106. Next, the SGLD matrix is built 
from the smoothed image 108. The SGLD matrix is a two-dimensional histogram 
corresponding to certain characteristics of the pixels in the input image. For each 
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pixel (m, n) in the smoothed image, a neighboring value is calculated using the 
following logic and equations: 
if |x(m, n + d) - x(m, n)| > |x(m + d, n) - x(m, n)| 
then y(m, n) = x(m, n + d), 

otherwise y(m, n) = x(m + d, n), (1), 
where x(m, n) is the smoothed pixel value at (m, n), (m, n + d) and (m + d, n) are 
vertical and horizontal neighbors, respectively, and d is a fixed integer (typically 1 
or 2). 

[0037] With reference to FIG. 2, a flowchart of an 

embodiment of the SGLD matrix initialization and construction process is shown. 

The initialization step 106 sets the SGLD matrix to zero (0) and sets a pixel 
counter (N) to zero (0) 154. The SGLD matrix is constructed from a low-pass 
filtered image 152 provided by the low-pass filter 104. Construction of the SGLD 
matrix begins by getting a pixel (m, n) 156 from the filtered image. A neighboring 
value for the pixel (m, n) is calculated using the algorithm in equation (1). If |x(m, 
n + d) - x(m, n)| > |x(m + d, n) - x(m, n)j 158, then y(m, n) = x(m, n + d) 160. 
Otherwise, y(m, n) = x(m + d, n) 162. As is apparent, if pixel (m, n) is in a flat area 
where x(m, n) is equal to y(m, n), the entry [x(m, n), y(m, n)] is on the diagonal. 

On the other hand, if (m, n) is on an edge, the difference between x(m, n) and y(m, 
n) will be significant, and [x(m, n) ? y(m, n)] will be far away from the diagonal. 

[0038] The entry [x(m, n), y(m, n)] in the SGLD matrix is 

then increased by one (1) and the pixel counter (N) is increased by one (1). Next, 
a check is made to determine if the calculation was for the last pixel 166 of the input 
image. If so, SGLD matrix construction is complete and the SGLD matrix is ready 
for feature extraction 168, Otherwise, the next pixel is retrieved 156 from the input 
image. 

[0039] For the matrix, the neighboring pixels in graphics 

images are expected to be either correlated or very different. In other words, for 
graphics images, SGLD matrix entries are usually either on the diagonal or far away 
from the diagonal This is because most pixels are either at the flat regions or on the 
edges. On the other hand, pixels of natural pictures are not expected to have many 
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abrupt changes. Accordingly, masses are expected to be concentrated at the entries 
that are near the diagonal for picture images. This shows the noisy nature of the 
picture images. 

[0040] Returning to FIG. 1, many features (e.g., variance, 

bias, skewness, fitness) can be extracted from the SGLD matrix to classify the input 
image between picture and graphics. The features can be implemented individually 
or combined in various methods (e.g., linear combination). Once the SGLD matrix 
is built, a feature or combination of features is selected for extraction 110 and 
processed using feature algorithms. For example, a first feature algorithm measures 
variance (V) (i.e., the second-order moment around the diagonal) 1 12 and is defined 
as: 

V = Z [n . mi > A s(m, n) (m - n) 2 /N (2), 

where s(m, n) is the (m, n)-th entry of the SGLD matrix, A is an integer parameter 

typically between 1 and 16 and; 

N = 2| n . m |> A s(m, n) (3) 

[0041] As the summation is over all (m, n) such that |m - n| 

> A, all the pixels in the flat regions are ignored. For graphics images, the remaining 
pixels are on the edges, while for picture images, both pixels in the noisy regions 
and pixels on the edges are included. Variance (V) is typically larger for graphics 
images than for picture images. 

[0042] The second feature algorithm measures average bias 

(B) 1 14 and is defined as: 

B= E| n _ m! > A s(m ? n) [n-\x(m)f /N (4) 5 
where ji(m) is the mean of s(m, n) for a fixed m. For a given m, the distribution of 
s(m, n) is roughly symmetrical about the diagonal for picture images, as noise 
typically has a zero mean symmetrical distribution. As a result B is usually small for 
picture images. For graphics images, s(m, n) is usually unsymmetrical and B is large. 

[0043] The third feature algorithm measures skewness (S) 

116 and is defined as: 



GL-l 
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c(n) = X s ( m > n ) and c = 2 c ( n )* ( 6 )* 



m=0 n=0 



[0044] The fourth feature algorithm measures fitness (F) 

1 18 and is defined to be: 



GL- 



F = fitness = • , (7)? 



a 

where a is defined such that: 

^ [s(/w, m + d) + s(m 7 m-d)] = 0.6xC. (8). 

d=0 

[0045] The image type decision 120 compares the result of 

the feature algorithm(s) to previously selected low and high thresholds (i.e., TL and 
TH, respectively) depending on the algorithm(s) and combinations selected. If the 
result of the feature algorithm(s) is below the low threshold (TL), the image is 
classified as a natural picture 122. If the result exceeds the high threshold (TH), the 
classification is synthetic graphics 126. Obviously, if the behavior of a particular 
feature is converse to this logic, the decision logic can be easily reversed to 
accommodate. If the result of the feature algorithm(s) is equal to or between the 
low and high thresholds, the class of the image cannot be determined (i.e., 
indeterminate 124) from the feature or combination of features selected. It is 
understood that a number of other alternatives are possible. For example, a result 
equal to a particular threshold can be said to be determinative of the image class, 
rather than indeterminate. Also, in certain circumstances the low and high threshold 
can be equal. 

[0046] With reference to FIG. 3, a flowchart of an image 

classification process using color discreteness features 200 in accordance with an 
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embodiment of the present invention is shown. The process 200 begins with an 
input image 202. First, the input image is transformed into a color space 204, in 
which the classification is performed. Although CIELUV space is used as one 
embodiment, many other color spaces can also be used. Next, the image is 
smoothed using an averaging filter 206 to remove any noise due to halftones. For 
example, a 4 x 4 filter was used successfully. Color histograms are computed for 
each of the three (3) color channels (i.e., luminance (L), U, and V) 208. The L, U, 
and V histograms are normalized 210 by the number of pixels in the image. The 
color representation scheme is invariant under rotation and translation of the input 
image and the normalization provides scale invariance. If (i) is the histogram of an 
image, where the index i represents a histogram bin, then the normalized histogram 
H is defined as follows: 

= (9). 

1=0 

[0047] Since graphics are generated using a limited number 

of colors, graphics images usually are comprised of a few areas of uniform color. 
Hence, the color histograms for a graphics image usually contain several sharp 
peaks. On the other hand, natural pictures usually contain more colors with 
smoothly varying transitions. Hence, natural pictures are more noisy and produce 
histograms containing fewer and smoother peaks. This difference in the histograms 
is captured in color discreteness algorithms for each color channel (i.e., R_L 
algorithm 212, R_U algorithm 214, and R_V algorithm 216). The color 
discreteness algorithms are defined as follows: 

GL-l 

R_L=X\H_L(i + l)-H_L(i)\ (10), 

GL-l 

R - U = Z\ H -U(i + l)-H_U(i)\ (11), 

1=1 
GL-l 

R - V = T\H_V(i + l)-H_V(i)\ (12), 

1=1 

where GL is the number of bins in the H_L, HJLI, and H_V color histograms 
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(typically, 256), 

[0048] The image type decision 218 compares the results of 

the color discreteness algorithms to previously selected thresholds (e.g., low 
threshold (TL) and high threshold (TH)). If the result of any color discreteness 
algorithm is above TH or below TL, the image is classified as either a graphics 224 
or picture 220 according to predetermined rules. Otherwise, the class of the image 
cannot be determined (i.e., indeterminate 222) by color discreteness features. 
Alternatively, the classifier may use all three (3) color discreteness features (as 
described above), any combination of two (2) features, or any one (1) feature. The 
color discreteness features can be computed faster than texture features (discussed 
above) or edge features (discussed below). 

[0049] With reference to FIG. 4, a flowchart of an image 

classification process using edge features 300 in accordance with an embodiment 

of the present invention is shown. The process 300 begins with an input image 302. 

First, edges of color areas in the image are detected 304 using a standard Canny 

edge detector and an edge map image is created. The parameters identified for the 

edge detector were determined empirically. Deviations that produce suitable results 

are also contemplated. Next, the edges in the edge map image are connected 306 

(e.g., using a standard 8-connected component algorithm). The average number of 

pixels per connected edge (E) in the edge map image is used as a feature 308. The 

algorithm for this edge feature is defined as: 

g _ No. of Edge Pixels 
No . of Connected Edges 

[0050] Typically, graphics have fewer connected edges, but 

each connected edge consists of a large number of pixels. On the other hand, 
pictures have a lot more connected edges, but usually very few pixels in each 
connected edge. This feature is particularly accurate for high values. In other 
words, if the value of E is high, it is almost certain that the image is graphics. 
However, if the value of E is low, nothing can be said about the image. This is 
because the E value may be low for graphics that have low frequency halftones or 
certain background. Accordingly, the image type decision 310 compares the result 
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of the feature algorithm to a previously selected high threshold (i.e., TH). If the 
result exceeds the high threshold (TH), the classification is synthetic graphics 3 14. 

Otherwise, the class of the image cannot be determined (i.e., indeterminate 3 12). 

It is understood that other alternatives are possible. For example, horizontal or 
vertical edges in the edge map may be used to classify images because the features 
are much more predominant in synthetic graphics than in natural pictures. Any 
combination of edge features or any one (1) edge feature can be used by the 
classifier. 

[0051] With reference to FIG. 5, a flowchart of an image 

classification process using a combination of SGLD texture features, color 
discreteness features, and edge features 400 in accordance with an embodiment of 
the present invention is shown. Notably, this image classifier combines all the 
features of the three (3) classifiers discussed above. SGLD texture, color, or edge 
features may be combined into one (1) classifier, whereby performance may be 
improved over classifiers using a single feature. 

[0052] While developing a classifier based a combination of 

texture, color, and edge features, it was observed that the classification and 
regression tree (CART) method, a public domain tree classifier, gave significant 
importance to the first color discreteness feature (R_L). It was also observed that 
the edge feature (E) was only accurate at large values (i.e., if the feature value was 
large) in determining that the image was a graphics. However, when the edge 
feature value was small, it was unable to determine whether the image was a picture 
or a graphics. All these observations can be combined in a rule-based tree classifier 
that uses a neural network at one (1) of its nodes. The combination of classifiers 
can analyze texture, color, and edge features to distinguish between picture and 
graphics images. 

[0053] The process 400 begins with an input image 102. 

Next, the features are extracted from the input image 404. Feature extraction 
includes compiling SGLD texture features 406 (e.g., variance (V), bias (B), 
skewness (S), fitness (F)), color discreteness features 408 (e.g., R_L, RJJ, R_V), 
and edge features 410 (e.g., pixels per connected edge (E), horizontal edges, 
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vertical edges). Alternatively, any combination of two (2) or more features that 
lead to the desired classification are contemplated, including the use of additional 
features. The SGLD texture features are compiled by performing steps 104-1 18 
of the process depicted in FIG. 1. Similarly, the color discreteness features are 
compiled by performing steps 204-216 of the process depicted in FIG. 3. Likewise, 
the edge features are compiled by performing steps 304-310 of FIG. 4. 

[0054] While developing the classifier, it was observed that 

the edge feature (E) was accurate at large values (i.e., when E is large, it is almost 
certain that the image is graphics). This observation was incorporated as a rule in 
the classifier. Hence, a first rule-based decision (i.e., E > TE 412) classifies the 
image as graphics 420, if; 

E > TE (14), 
where TE is a previously identified high threshold value for the edge feature. 
Experimentally, TE = 120 produced satisfactory results. 

[0055] It was also observed that the public domain tree 

classifier CART gave significant importance to the first color discreteness feature 
(R_L). This observation was also incorporated as a rule in the classifier. Hence, 
a second rule-based decision (i.e., R_L > TH, R_L < TL 414) classifies the image 
as a graphics, if: 

R_L>TH ( i 5)) 
and as picture, if: 

RJL<TL (1 6)> 
where TH and TL are high and low threshold values, respectively, for the R_L color 
discreteness feature. Experimentally, TH = 0.15 and TL = 0.05 produced 
satisfactory results. 

[0056] If the class of the image cannot be determined from 

the rules the neural network 416 operates using any combination of two or more of 
the texture, color, and edge features to make the determination. The features are 
scaled to [0, 1] before feeding into the neural network. One embodiment of the 
neural network is a standard feedforward architecture. A back-propagation 
algorithm is implemented for training the network. The feedforward architecture 
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includes an input layer, a hidden layer, and an output layer. The input layer includes 
a plurality of source nodes (e.g., eight (8)). The hidden layer and the output layer 
are each comprised of one (1) neuron (i.e., computation nodes). The source nodes 
are projected onto the computation nodes, but not vice versa - hence the "feed 
forward" name. The hidden neuron intervenes between the external input and 
output layers and enables the network to extract higher-order statistics. 

[0057] The back-propagation algorithm, also known as the 

error back-propagation algorithm, trains the neural network in a supervised manner. 
Basically, back-propagation learning consists of two (2) passes through the 
different layers of the network: a forward pass and a backward pass. In the forward 
pass, an input pattern is applied to the source nodes and its effect propagates 
through the network. The output produced represents the actual response of the 
network. During the forward pass the synaptic weights of the network are all fixed. 
During the backward pass, on the other hand, the synaptic weights are all adjusted 
in accordance with an error-correction rule. Specifically, the actual response of the 
network is subtracted from a desired (target) response to produce an error signal 
This error signal is then propagated backward through the network, against the 
direction of synaptic connections - hence the name "error back-propagation." The 
synaptic weights are adjusted to make the actual response of the network move 
closer to the desired response in a statistical sense. 

[0058] As shown in FIG. 5, the neural network has eight (8) 

inputs 404 (i.e., V, B, S, F, RJL, R_U, R_V, E) and one (1) binary output (i.e., 
picture/graphics 422). The rule-based portion of the classifier (i.e., 412, 414) does 
not need any training. The neural network 416 was trained with samples that were 
already classified correctly by the rule-based classifier portion and tested on the rest 
of the samples. 

[0059] With reference to FIG. 6, a block diagram of an 

image segmentation system 500 using a "binary" image classification process (i.e., 
classification of images between picture or graphics classes) is shown. The 
picture/graphics classifiers (i.e., 100, 200, 300, 400) of FIGS. 1-4 are "binary" 
classifiers and could be implemented in such a system 500. As described above for 
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FIGS. 1-4, an input image 502 is provided to a feature extractor 504. The feature 
extractor 504 extracts pertinent characteristics (i.e., features) based on the 
parameters required by algorithms of the binary classifier 506. The binary classifier 
506 exercises algorithms designed to classify the input image between a natural 
picture or a synthetic graphics image (e.g., [0, 1] where 0 indicates picture and 1 
indicates graphics). This binary classification result is provided to a switch 508. 
The switch 508 receives the input image 502 and switches it between picture 
processing 510 and graphics processing 512, depending on the binary classification 
result. Picture processing 510 processes the image in a manner tailored to maximize 
the quality of natural picture images (e.g., gamut mapping). Similarly, graphics 
processing 512 is tailored to maximizes the quality of synthetic graphics images 
(e.g., filtering). If the input image is classified as a picture, the input image 502 is 
switched to picture processing 510 and a picture output 514 is produced. 
Alternatively, if the image is classified as graphics, the input image 502 is switched 
to graphics processing 5 12 and a graphics output 516 is produced. In the event that 
the binary classifier 506 cannot determine the class of the input image, one (1) of 
the processes (e.g., picture processing 510) may be selected by default. 

[0060] With reference to FIG. 7, a block diagram of an 

image processing system using a "soft" image classification process (i.e., 
classification of image areas between picture, graphics, or fuzzy classes) and an 
associated process for blending image processing functions based on the 
classification is shown. 

[0061] The "soft" fuzzy image classification is an 

improvement over the fuzzy classification process (e.g., as disclosed in U.S. Patent 
No 5,778,156 to Schweid) by making the classification decision "soft." This is done 
by using a neural network, with image features as inputs and "two" outputs. The 
soft classification result is then used to "blend" the down stream image processing 
functions (i.e., gamut mapping or filtering). It can also be used to evaluate the 
confidence level of the classification, and take appropriate actions. Again, as 
described above for FIGS. 1-4, an image input 602 is provided to a feature 
extractor 604. The feature extractor 604 extracts two (2) or more pertinent 
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characteristics (i.e., features) from the input image 602 and provides it to a soft 
classifier 606 (e.g., neural network, fiizzy decision tree, Gaussian maximum 
likelihood, or any classifier with continuous, rather than binary output). As 
discussed above for binary classifiers, the features provided to the classifier can be 
indicative of various distinguishing characteristics of an input image. For example, 
two (2) or more texture (e.g., V, B, S, F), color discreteness (e.g., RJL, R_U, 
R_V), or edge (e.g., E) features can be implemented in any combination. 
Additional features that lead to the desired classification are also contemplated. 

[0062] In one embodiment, the soft classifier 606 is a neural 

network in a standard feedforward architecture, similar to the neural network 
described above in reference to FIG. 5. However, in the neural network of the 
"soft" classifier, the hidden layer includes one (1) or two (2) neurons and the output 
layer is comprised of two (2) neurons. Like the neural network above, a back- 
propagation algorithm is implemented for training the network. Each of the two (2) 
outputs (i.e., a, b) of the neural network will have a value that ranges between a 
minimum and a maximum (e.g., between 0 and 1). The output value represents the 
level of membership for an area of the input image in each of two (2) classes (e.g., 
picture, graphics). Ideally, when an area is in the graphics class, the output will be 
[1,0]. Conversely, if the area is in the picture class, the output will be [0, 1]. In 
actual cases using these rules, both outputs (e.g., [a, b]) will usually range between 
0 and 1, indicating that the area of the input image is in the fuzzy class and further 
indicating the level of membership to both picture and graphics classes. The "soft" 
classification result 608 (i.e., an input image with picture, graphics, and/or fiizzy 
areas) is used to "blend" 610 the down stream image processing functions (e.g., 
image processing 1 (612), image processing 2 (614)), creating a "blended" image 
processing function, to produce an output image 616. In the preferred embodiment, 
image processing 1 is a gamut mapping/filtering process for pictures and image 
processing 2 is a gamut mapping/filtering process for graphics. However, 
alternative configurations are envisioned with additional image processing functions 
or different functions. 
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[0063] In another embodiment, the input image 602 is 

provided to each of a plurality of image processing functions (e.g., image processing 
1 (612) and image processing 2 (614)), rather than to the "blender," This is shown 
in FIG. 7 via dashed lines. In this alternative, the "soft" classification result 608 
(i.e., an input image with picture, graphics, and/or fuzzy areas) is used to "blend" 
610 the processed images resulting from the multiple image processing functions 
to produce a "blended" output image 616. 

[0064] Alternatively, if a binary decision is desired, a and b 

are compared to make the classification decision. The difference between a and b 
provides the classification based on the following rules: 
a-b » 0, graphics class (17); 
a-b = 0, indeterminate (18); and 

a-b « 0, picture class (19). 

[0065] The difference between a and b can also be used as 

a confidence level of the classification based on the following rules: 
a-b » 0, strong confidence of graphics class, little confidence of picture class; 
(20); 

a-b = 0, uncertainty in classification; (21); and 

a-b « 0, strong likelihood of picture class, little confidence of graphics class (22). 

[0066] In some spatial gamut mapping techniques, a spatial 

feedback filter is used to preserve luminance variations in the gamut mapping 
process. The optimal footprint and coefficients of the filter depend heavily on the 
nature of the image content (i.e., natural picture versus synthetic graphics). Where 
this technique is implemented by blending 601, the output of the soft classifier 608 
can be used to steer the filter parameters. Similarly, methods of blending filter 
coefficients have been described in US Patent 5,778,156 to Schweid et al. entitled 
"Method and System for Implementing Fuzzy Image Processing of Image Data." 

[0067] Even if downstream image processing functions 

(e.g., gamut mapping or filtering) are not blended 610, the "soft" classification 
result 608 can be used to bias the classification decision to be on the safe side or to 
select a safe or neutral position when the confidence level is low. 
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[0068] The invention has been described with reference to 

the preferred embodiments. Obviously, modifications and alterations wall occur to 
others upon reading and understanding the preceding detailed description. It is 
intended that the invention be construed as including all such modifications and 
alterations insofar as they come within the scope of the appended claims or the 
equivalents thereof. 



-21 - 



